3D Pose Estimation of the Face from Video

Face pose information is valuable for a variety of applications including unconstrained face recognition, natural human computer interfaces, and video database indexing. 3D pose estimation is a critical requirement for accurate face recognition using view varying representations, such as 2D intensity images. 3D pose extraction in this context requires 3D information, which is present in image sequences if we assume the moving objects in the sequence are primarily rigid. This paper presents a motion based pose estimation system which computes the 3D pose of the head in each frame of a video sequence. Generic low level features, such as corners, are identified and tracked in the video stream. The feature tracks are processed by a shape from motion algorithm which produces estimates of 3D geometry and pose. The geometry and pose estimates are considered together with facial structure constraints, temporal constraints, and initial pose estimates to refine knowledge of the specific face structure and its pose. We describe this system and show that it works on human faces. This is significant because the face has many smooth surfaces which make it difficult to extract dense intensity features. In Face Recognition: From Theory to Applications, H. Wechsler, P. J. Phillips, V. Bruce, F. Fogelman Soulie, and T. Huang (Eds.), NATO ASI Series F, Springer-Verlag, 1998.